Methods and apparatus for the creation and transmission of 3-dimensional images

ABSTRACT

Three-dimensional color images are produced by combining the red image plane from a left color camera and the blue and green image planes from a right color camera. Techniques for compensating for over or underexposure in a particular image plane are deployed as well as techniques for minimizing subjective disturbance when viewing relatively pure color regions of a 3-dimensional image and for transmission of 3-dimensional color television images to users.

This application is a division of application Ser. No. 08/335,381 filedNov. 3, 1994, still pending.

TECHNICAL FIELD

This invention relates generally to the field of image processing andmore particularly to the creation, enhancement transmission, andpresentation of 3-dimensional (3-D) images on a 2-dimensional viewingsurface.

BACKGROUND ART

Since the invention of the stereoscope in 1847, there has been a desirefor emulating the 3-D images of nature instead of being content with twodimensional images which lack realism due to the absence of depth cues.Many techniques have been devised and developed for producing 3-Dimages, each varying in degree of success and quality of image. Thesetechniques generally belong to two major classes, namely theautostereoscopic imaging class which produces 3-D images which can beviewed freely without spectacles, and the binocular stereoscopic imagingclass which produces 3-D images which requires observers to wearspectacles or viewers. Techniques of the later class have been found in3-D movies of the 1950's and in occasional 3-D image productions such as3-D comic books.

Color separation of stereo images has been utilized for over fifty yearsin the production of photographs, 3D movies and the printed page. Inprior art devices such as shown in U.S. Pat. No. 3,712,119, stereoimages are separated by mutually extinguishing filters such as ablue-green lens filter over one eye and a red filter over the other eye.With this combination, a full true color image is not obtained, and thiscolor combination may cause eye fatigue, and color suppression.

In the prior art an object of a single pure color matching the filtercolor e.g. red or blue-green, would be transmitted only to one eye andso would not appear in stereo. However, pure colors are rare, and mostobjects are off-white, or pastel shades and so contain all three primarycolors. Thus, most objects will have some component of each color andthis enables the separation of right and left stereo images.

Prints, drawings or representation that yield a 3-D image when viewedthrough appropriately colored lenses are called anaglyphs.

An anaglyph is a picture generally consisting of two distinctly colored,and preferably, complementary colored, prints or drawings. Thecomplementary colors conventionally chosen for commercial printings ofcomic books and the like are orange and blue-green. Each of thecomplementary colored prints contains all elements of the picture. Forexample, if the picture consists of a car on a highway, then theanaglyph will be imprinted with an orange car and highway, and with ablue-green car and highway. For reasons explained below, some or all ofthe orange colored elements of the picture are horizontally shifted invarying amounts in the printing process relative to their correspondingblue-green elements.

An anaglyph is viewed through glasses or viewers having lenses tintedabout the same colors used to prepare the anaglyph (hereinafter, "3-Dglasses"). While orange and blue-green lenses are optimally used with anorange and blue-green anaglyph, red and blue lenses work satisfactorilyin practice and apparently are conventionally used.

The orange elements in the picture are only seen through the blue lens,the red lens "washing out" the orange elements. For the same reason, thegreen-blue elements are only seen through the red lens. Hence, each eyesees only one of the two colored pictures. But because the differentcolored elements are horizontally shifted in varying amounts, theviewer's eyes must turn inward to properly view some elements, and turnoutward to properly view others. Those elements for which the eyes turninward, which is what the viewer does to observe a close object, arenaturally perceived as close to the viewer. Elements for which theviewer's eyes turn outward are correspondingly perceived as distant.Specifically, if the blue lens covers the viewer's right eye, as isgenerally conventional, then any blue-green element shifted to the leftof its corresponding orange element appears to the viewer as close. Theelement appears closer the greater the leftward shift. Conversely, as agreen-blue element is shifted only slightly leftward, not at all, oreven to the right of its corresponding red element, that element willappear increasingly more distant from the viewer.

In addition to horizontally shifting the element pairs relative to eachother, some users of anaglyphy for comic books also vertically shift theelement pairs a slight amount relative to each other. Those usersbelieve that the slight vertical shift improves the 3-D effect.

Normally 3-D images appear monochromatic when viewed through 3-Dglasses.

Three dimensional techniques are closely related to the psychology andphysiology of an observer's cognitive processes. Subtle changes inselection of portions of the spectrum presented to each eye can resultin significant changes in the observer's perception. Even when viewingthe same 3-dimensional image through the same viewers, differentobservers may perceive a 3-dimensional image in different ways.

One problem common to most observers arises when viewing a pure red orpure blue region of a 3-dimensional image through red/blue 3-dimensionalglasses. In such circumstances, one eye will perceive black and theother eye will perceive nothing. This has a psychological and/orphysiological impact on the viewer which most viewers find disturbing.

Further, when observing 3-dimensional images in which the left and rightimages are captured using complementary filters, the images reproducedin the colors of the filters, and viewed through viewers of the samecolors (e.g. red/blue glasses) which separate the images, 3-dimensionalimages appear only in black and white. That is, color information islost in the preparation of the 3-dimensional image. This ischaracteristic of most 3-dimensional images.

When processing color images using computers, it is common to separatean image into (e.g.) red, green and blue image components. Commonly eachcolor component is referred to as an image plane. In the display ofcolor images on a color cathode ray tube it is common to applyinformation from each color image plane to a respective electron gun ofthe cathode ray tube.

Normally, in the past, when preparing 3-dimensional motion pictures, theanaglyph frames were prepared in the post production suite.

When color images are captured, it sometimes occurs that one of thecolors utilized for representing the image may be overexposed orunderexposed as reflected, inter alia, in an inadequate dynamic rangefor that color. That is, anytime the color appears at all, it appears atmaximum value or anytime it appears it appears at some minimum valueinstead of being spread over the entire dynamic range of representation.This adversely affects the quality of 3-D image produced

The prior art generally required complex specialized equipment for thetransmission of 3-dimensional images. This inhibited the use of 3-Dtechnology because much capital investment has been devoted to equipmentfor handling regular 2-dimensional images. It would be desirable toutilize 2-dimensional transmission equipment to produce 3-dimensionalimages.

DISCLOSURE OF THE INVENTION

Accordingly, one advantage of the invention is the creation of3-dimensional images which are perceived in color.

Another advantage of the invention is the elimination of the subjectivedisturbance perceived when either pure red or pure blue portions of animage are viewed.

Another advantage of the invention relates to correcting overexposure orunderexposure of a particular color utilized in creation of3-dimensional images.

Another advantage of the invention is the creation of 3-dimensionalmoving images on line rather than in the post production suite.

Another advantage of the invention is the transmission of 3-dimensionalcolor television images over existing broadcast and communicationfacilities in a cost effective manner.

According to the invention, the foregoing and other objects andadvantages are obtained by providing a device for making 3 dimensionalcolor images which uses a left and a right color video camera, each ofwhich produces synchronized outputs comprising 3 image planes with eachplane corresponding to red, green and blue color informationrespectively. The red image plane from the left color video camera iscombined with the green and blue image planes from the right color videocamera to produce a three dimensional output signal. A frame grabber canbe used for each color video camera to capture related frames of eachcolor video camera to produce a still three dimensional digital anaglyphof the images captured by the frame grabber.

Another aspect of the invention involves a method for making threedimensional color images of a scene using a left and a right color videocameras and producing an output video signal from each camera havingred, green and blue image planes and by providing green and blue imageplanes from the right color video camera and a red image plane from theleft color video camera as the output signal.

Another aspect of the invention relates to apparatus for making threedimensional images using a left and a right video camera each producingan output comprising 3 image planes, each plane corresponding toparticular color information. One image plane from one of the left orright video cameras, preferably the red image plane, is replaced with animage plane from the other of the left or right video cameras and theinformation from two cameras is combined into one or more threedimensional video images.

Another aspect of the invention relates to a method of creating a threedimensional image from two images captured from two different positionsby resolving each of the two images into three separate color componentsand combining one color component from one of the two images with twocolor components from the other of the two images to create a combinedimage.

Still another aspect of the invention relates to a method of creating athree dimensional image from two digital images captured from twodifferent positions by resolving each of the two images into separatecolor planes, and combining one color plane from one of the two imageswith at least one color planes from the other of the two images tocreate a combined three dimensional image.

A different aspect of the invention relates to a method and apparatusfor making three dimensional images in which a left and a right colorvideo camera each produce an output comprising red, green and blue imageplanes. Green and blue image planes from the right color video cameraand the red image plane from the left color video camera are combinedinto an output signal. The color value of each pixel of the outputsignal is monitored and, when the value lacks a first threshold amountof blue or green color or lacks a second threshold amount of red orgreen color, a quantity of blue and/or green color or a quantity of redand/or green color, respectively, is added to the pixel color value. Asa result, information from two color video cameras is combined intothree dimensional color video images which are perceived more acceptablywhen observed through red/blue viewers.

Another aspect of the invention involves method and apparatus for makingthree dimensional images by using a left and a right color video camerato produce an output comprising red, green and blue image planesrespectively. Green and blue image planes from the right color videocamera and a red image plane from the left color video camera arecombined as an output signal. The color values of each pixel of theoutput signal is monitored and, when the red value of a number of pixelsindicates underexposure or overexposure in red, substituting abrightened value from the green image plane of the left color videocamera. As a result, information from two color video cameras iscombined into three dimensional color video images which are perceivedmore acceptably when viewed through red/blue viewers.

The invention also relates to a method and apparatus for broadcastingthree dimensional television images by capturing images using a left anda right color television cameras. Each camera produces an outputcomprising red, green and blue image planes. A signal containing theoutput from the left color video camera is transmitted using a firsttelevision transmitter and a signal containing the output from the rightcolor video camera is transmitted using a second television transmitter.At a receiver, signals from the first and second television transmittersare received and respective outputs comprising the three image planesfrom the left and right color video cameras, produced. Green and blueimage planes from the right color video camera and a red image planefrom the left color video camera are combined into an output signal forviewing on a display.

Another aspect of the invention related to a method and apparatus forbroadcasting three dimensional television images by producing a colorvideo output comprising red, green and blue image planes from left and aright color video cameras; broadcasting signals containing the outputfrom the right color video camera using a television transmitter;transmitting signals containing the red image plane from the left colorvideo camera over a point to point communications link; and receiving ata receiver the signals from the television transmitter and said signalsfrom the point to point communications link for providing green and blueimage planes from the right color video camera and a red image planefrom the left color video camera as an output signal to a display forviewing. Thus, information from two color video cameras is combined intothree dimensional color video images at a receiver.

Another aspect of the invention includes the computer generation of3-dimensional anaglyphs. An object is represented in a 3-dimensionalrepresentation such as a wire frame generated using a 3-dimensionaldatabase. A full color surface can be rendered (applied) on the wireframe. Two different 2-dimensional views of the object are generatedfrom different perspectives, corresponding, in one example, to viewsfrom the left and right eyes. The red image plane from the leftperspective is combined with the blue-green image planes from the rightperspective to create a 3-dimensional anaglyph representation of theobject.

Still other objects and advantages of the present invention will becomereadily apparent to those skilled in the art from the following detaileddescription, wherein only the preferred embodiment of the invention isshown and described, simply by way of illustration of the best modecontemplated of carrying out the invention. As will be realized, theinvention is capable of other and different embodiments, and its severaldetails are capable of modifications in various obvious respects allwithout departing from the invention. Accordingly, the drawings anddescription are to be regarded as illustrative in nature and not asrestrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a common way of storing color pixelinformation for digital processing in the prior art.

FIG. 2 is an illustration of how image planes from two video cameras canbe combined to produce a color 3-D image.

FIG. 3 is a flow chart of image processing required to produce a color3-dimensional image from two 2-dimensional colored digital images.

FIG. 4 is a drawing illustrating the creation of a 3-dimensional colorstill image.

FIG. 5 is a functional illustration of how 3-dimensional color imagedata is processed to achieve a more pleasing color presentation.

FIG. 6 is a functional depiction of how over- or underexposure of animage plane can be corrected.

FIG. 7 is a block diagram of a system for transmitting and receiving3-dimensional television images.

FIG. 8 is a block diagram of another system for transmitting andreceiving 3-dimensional television images.

BEST MODE FOR CARRYING OUT THE INVENTION

The invention is based in part upon a recognition that color3-dimensional images can be produced by shifting color planeinformation. FIG. 1 illustrates how digital color image information fora single pixel may be stored when undertaking digital image processing.Twenty-four bits of information are illustrated in FIG. 1, however, thenumber of bits per pixel and the number of bits per field may beselected to meet the needs of a particular design. In the exampleillustrated in FIG. 1, 8 bits illustrated at 100 represent redinformation whereas the two sets of 8 bits each at 110 and 120represents color intensity levels for respective green and blue colorcomponents for a particular pixel. With 8 bits each, each indication ofcolor intensity level may represent 256 different intensity levels. The8 levels in each of 3 colors permits 2²⁴ color intensity combinations tobe represented.

In a typical cathode ray tube (CRT) shadow mask display, separateelectron guns are utilized to activate separate red, green and bluephosphor dots on the screen selectively. The red, green and blue bits ofthe color information depicted in FIG. 1 are utilized to establish theintensity of red, green and blue components or in other words the colorfor the pixel. If a pure blue pixel were desired, the red and greenelectron guns would be turned off and only the blue gun would bombardthe blue phosphor dot of the triad at an intensity level set by the 8bit intensity level value for blue (120). If a pure red pixel weredesired, the green and blue guns would be turned off by virtue of the 0values represented in fields 110 and 120 of FIG. 1, and the red gunwould be activated at the intensity level set by the 8 bit field 100.For mixed colors, the intensity levels from the three fields 100, 110and 120 control the color and intensity of the light output from aphosphor dot triad in the CRT screen.

Digital image arrays can be very large. For example, digitizing a fairlystandard black and white photographic image can involve a storagerequirement of 8 megabits for an array of 1024×1024 elements. Thisassumes 256 intensity levels. One byte (8 bits) of data is utilized torepresent the intensity level of each pixel.

In the example shown in FIG. 1, 8 bits are utilized to represent eachcolor intensity level. Thus a color image of 1024×1024 elementsutilizing the storage scheme of FIG. 1, would require 24 megabits ofstorage. In many applications, even finer resolution is required withthe attendant increasing storage overhead.

In the storage of a 1024×1024 color image utilizing the scheme shown inFIG. 1, 8 megabits of information constitutes red only information, aseparate 8 megabits of information constitutes green only informationand a final 8 megabits of information constitutes blue only information.The aggregation of storage dedicated to particular color information iscalled a color plane.

The system described above is referred to typically as RGB (red, green,blue) color storage.

Psychologists usually describe color by "hue" the color name whichdepends on average wave length, by "saturation", the purity of thecolor, and by "intensity", the overall brightness of the color. Standardformulas are well known in the art for converting between RGBrepresentation and HSL representation.

YIQ coding used in commercial television transmission utilizes adifferent transformation representation than the HSL system. It requiresa luminance component (Y) for monochrome sets and two chromanancecomponents (IQ) which take weighted differences of the RGB values.

FIG. 2 illustrates a particularly simple technique for generating3-dimensional images in which the color information is retained. Leftand right color video cameras 200 and 210 are positioned so as tocapture two different views of the same scene. Preferably, the opticalaxis of the objective lenses of left and right video cameras 200 and 210are separated by approximately the interocular distance. Each left andright color video cameras 200 and 210 have respective red, green andblue outputs which are labeled R_(L), G_(L) and B_(L) and R_(R), G_(R)and B_(R), respectively. The red, green and blue outputs from each ofthe two cameras is fed to color planes switch 220 where the red colorplane from the left camera is substituted for the red color plane of theright camera to produce a composite output R_(L), G_(R) and B_(R) atoutputs 230, 240 and 250 respectively. If analog, these signals may besampled directly and stored as separate color planes. If digital, theymay be combined into the format shown in FIG. 1. Alternatively, analogoutput from 230, 240 and 250 may be used to produce a 3-dimensionalcolor video image.

FIG. 3 shows a flow chart of image processing required to produce acolor 3-dimensional image from two color digital images. First tworelated color images are captured (300) and optionally stored as twocolor digital images (310). Then, one color plane from one image isreplaced with the color plane from the other image to create a3-dimensional result in image (320) which can be either stored in animage store or viewed on a display, respectively (330).

FIG. 4 illustrates how a 3-dimensional color still pictures, or ananaglyphs, may be created. Color digital cameras 400 and 410 capturerelated still images of a scene to be photographed. Again, it ispreferred that the optical axis of the objective lenses of the left andright digital cameras be separated by approximately the interoculardistance. Color pixels in digital format are output from cameras 400 and410 on lines 420 and 430, respectively and are fed to respective colorplane separators 440 and 450 respectively. The output of the color planeseparators are respectively combined in color plane combiner 460 suchthat the red image plane from the right camera is replaced with a redimage plane from the left camera in color plane combiner 460. The resultis an output, 470, which directly produces a color anaglyph of the imagecaptured by the cameras in a particularly simple and efficient manner.

FIG. 4 also illustrates the computer generation of 3-dimensionalanaglyphs. An object is represented in a 3-dimensional representationsuch as a wire frame generated using a 3-dimensional database. A fullcolor surface can be rendered (applied) on the wire frame. Two different2-dimensional computer generated views of the object are created fromdifferent perspectives, corresponding to views from the left and rightcameras shown in FIG. 4. The red image plane from the left perspectiveis combined with the blue-green image planes from the right perspectiveto create a 3-dimensional anaglyph representation of the computergenerated object.

FIG. 5 is an illustration of functionality utilized to eliminate thedisturbing effects of viewing pure blue or pure red colors throughred/blue viewers. The unmodified 3-D color information such as producedat the output of the FIG. 2 circuitry, is applied at the input puritymonitor 500. Purity monitor 500 monitors color plane information anddetects when a pixel is either pure blue or pure red within certaintolerances. If a pure red pixel is detected, a certain quantity ofgreen/blue information is added by color adder 510. Similarly, if a pureblue pixel is detected, a certain amount of red/green information isadded. The net result is to eliminate pure red or pure blue pixels andthe psychological/physiological disturbances that come from viewing suchpixels with red/blue viewers. The modified 3-D color output informationfrom color adder 510 has somewhat more muted color information, but theoverall subjective viewing quality is improved by the processingillustrated in FIG. 5.

FIG. 6 is a functional illustration of correction of overexposure orunderexposure in a particular image plane. The inputs to exposuremonitor 600 are unmodified 3-dimensional color information such as thatgenerated by the circuitry of FIG. 2. Since red over-or-under-exposurehas a significant impact on 3-dimensional image color quality, FIG. 6illustrates monitoring of the red color plane for under or overexposure.Exposure monitor 600 detects over or underexposure by essentiallycreating a histogram of the red color values being received by theexposure monitor. If most color values are at the high end, red islikely over-exposed. If most values are at the low end, red is likelyunderexposed. When such a circumstance is detected, exposure correctioncircuit 610 responds by substituting a brightened version of the greenimage plane from the same camera from which the red image planeoriginated for the over or underexposed red image plane.

Exposure monitor 600 periodically samples the incoming values from thered image plane and counts the number of pixels at each level ofintensity. At the end of the sampling duration, the number of samples ateach of the upper and lower ends of the red color value are examined todetermine whether an inordinate percentage of color values fall at thoseextremes. If the number of pixels at either extreme is inordinatelyhigh, then output signal 620 or 630 is generated as appropriate. Thesetwo output signals are OR'd in gate 640 and control an electronic switchwhich replaces the red image plane with a brightened version of thegreen image plane from the same camera which originated the red imageplane. Such brightening occurs by incrementing the green intensity levelby "N" steps.

Thus, intervals of under or over-exposure of the red image plane can becorrected to provide a pleasing color 3-dimensional image to the viewer.

FIG. 7 illustrates a technique for transmitting 3-dimensional images toa viewer utilizing standard television broadcasting equipment. The colorvideo cameras 710 and 720 capture the left and right views respectivelyof a scene to be transmitted. The RGB output from camera 710 and 720 arefed to respective color television transmitters 730 and 730' whichtransmit on separate television broadcast channels. Although a singleantenna 740 is shown for these transmitters, each transmitter may haveits own separate antenna and, in fact, each transmitter may be locatedat a separate physical location. Images from the two transmitters arereceived at antenna 750 and fed to receivers 760 and 760' where the RFinformation is converted to color video baseband in decoder 770 and 770'and synchronized RGB outputs are provided by the decoders. Receivers 760and 760' may have individual antennas instead of a common antenna 750 asshown. The RGB outputs of the decoder 770 and 770' are connected asshown so that the red image plane is taken from the left decoder and thegreen and blue image planes are taken from the right decoder and the RGBinformation is applied to a display such as television set 780.Alternatively, the full left and right images could be applied to leftand right displays of a virtual reality viewer, where the left eye wouldsee the full left image and the right eye, the full right image.

Thus, using two channels of normal broadcast television equipment onecan create 3-dimensional color television at a user location.

FIG. 8 is an illustration of another method of providing 3-dimensionalcolor television images to a user. One color camera 810 captures a scenenormally and its color output is broadcast over color televisiontransmitter 820 and antenna 830 to a receiving antenna 850 and areceiver 840 tuned to the channel. Right decoder 860 produces RGB outputfrom the signal received by the receiver and the green and blue imageplanes are connected to the display of television set 870. The red imageplane from left camera 800 is transmitted over a point to pointcommunication link represented by network 880 to the viewer's locationwhere it may be combined with synchronized green and blue image planesfrom the right decoder 860 to produce a 3-dimensional image on thedisplay of television receiver 870. A simple switch 890 allows a user toswitch between normal color television reception and 3-dimensional colortelevision reception. In normal mode all three image planes from rightdecoder 860 are fed to color television display 870. In 3-dimensionalcolor television mode, a user might dial up the video informationprovider over network 880 and request 3-dimensional service. The videoinformation provider would then allow, after suitable arrangements forcompensation have been made, the user to download the red image plane ofthe left camera 800 to complete the 3-dimensional image. The presence of3-dimensional image data on the point to point communications link canbe detected automatically and utilized to control switch 890 so that3-dimensional information is displayed when 3-dimensional information isavailable and otherwise 2-dimensional information is displayed.

Accordingly, there have been described methods and apparatus for thecreation and transmission of 3-dimensional color images which overcomethe problems of the prior art. In this disclosure, there has been shownand described only the preferred embodiment of the invention, but, asaforementioned, it is to be understood that the invention is capable ofuse in various other combinations and environments. It is capable ofchanges or modifications within the scope of the inventive concept asexpressed herein.

What is claimed is:
 1. Apparatus for making three dimensional imagescomprising:a. a left and a right color video camera each producing anoutput comprising 3 image planes each plane corresponding to red, greenand blue color information respectively, b. a combiner for providinggreen and blue image planes from one of the left or right color videocameras and a red image plane from the other color video camera as anoutput signal, c. a monitor detecting the color values of each pixel ofthe output signal, and d. an exposure corrector substituting abrightened value from one of the green or blue image planes of the othercolor video camera for the corresponding values in the red image planeused to create the output signal when the monitor detects that the redvalue of a number of pixels indicates underexposure or overexposure inred, whereby information from two color video cameras is combined intothree dimensional color video images.
 2. A method for making threedimensional color images of a scene comprising:capturing said sceneusing a left and a right color video cameras; b. producing an outputvideo signal from each camera comprising 3 image planes, each planecorresponding to red, green and blue color information respectively; c.producing a three dimensional output signal by providing green and blueimage planes from one of the left or right color video cameras and a redimage plane from the other color video camera as the output signal; andd. monitoring the color values of each pixel of the output signal and,when the red value of a number of pixels indicates underexposure oroverexposure in red, substituting a brightened value from one of thegreen or blue image planes of the other color video camera for thecorresponding values in the red image plane used to create the outputsignal, whereby information from two color video cameras is combinedinto three dimensional color video images.